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Fught  Service  Specialist  Initial  Qualificatkm>is  Course: 
Content  Validation  oe  FAA  Academy  Course  50232 


INTRODUCTION 

Purpose  of  FAA  Academy  course  50232 
The  purpose  of  the  FAA  Academy  course  50232, 
Flight  Service  Specialist  Initial  Qualiflcations 
Course,  is  to  predia  a  student’s  probability  to  suc¬ 
ceed  to  the  full  performance  level  as  a  flight  service 
specialist  (FAA  Academy,  1989,  p.  iii).  The  pro¬ 
posed  Course  Management  Guide  asserts  that 

“Students  who  fail  to  obtain  a  passing 
grade  upon  completion  of  the  Flight  Ser¬ 
vice  Initial  Training  will  be  identified  as  a 
failure  and  withdrawn  from  training.  ... 
Probationary  students  who  fail  the  course 
will  be  terminated  by  the  Aeronautical 
Center  or  may  elect  to  resign...” 

Legal  requirements  for  validation 
Federal  Aviation  Administration  (FAA)  policy 
requires  compliance  with  all  appropriate  federd 
policies,  guidelines,  and  procedures  to  assure  equal 
opportunity  in  employment  (FAA,  1985).  The  ap¬ 
propriate  federal  guidelines  in  this  case  are  the 
Equal  Employment  Opportunity  Commission  (1978) 
Uniform  Guidelines  on  Employee  Selection  Pro¬ 
cedures.  The  scope  of  the  Guidelines  are  defined  in 
section  l.B  as  follows: 

“Employment  decisions.  These  guidelines 
apply  to  tests  and  other  selection  procedures 
which  are  used  as  abasis  for  any  employment 
decision.  Employment  decisions  include  but 
are  not  limited  to  hiring,  promotion,  demo¬ 
tion,  membership  (for  example,  in  a  labor 
organization),  referral,  retention,  and  licetts- 
ing  and  certification ...” 

Purpose  of  this  study 

As  is  clear  from  the  proposed  Course  Manage¬ 
ment  Guide,  employment  decisions  for  flight  ser¬ 
vice  specialists  will  be  made  on  the  basis  of  their 
performance  in  FAA  Academy  course  50232.  That 
initial  training  course  is  therefore  a  selection  proce¬ 
dure,  as  defined  by  the  Guidelines,  and  must  be 
validated  for  use  in  maldng  employment  decisions. 
The  purpose  of  this  post-hoc  evaluation  of  the  con¬ 
tent  of  course  50232,  therefore,  was  to  assess  the 
validity  of  FAA  Academy  course  50232  for  making 
personnel  decisions  in  the  flight  service  sp<*cialist 
occupation. 


Technical  standards 

This  report  is  the  first  of  a  series  of  studies  that 
will  be  required  to  establish  the  validity  of  FAA 
Academy  course  50232  in  accordance  with  relevant 
legal,  technical,  and  professional  standards.  These 
standards  include  the  Uniform  Guidelines  on 
Employee  Selection  Procedures  (Equal  Employ¬ 
ment  Cipportunity  Commission,  and  others,  1978), 
the  Principles  for  the  Validation  and  Use  of  Per¬ 
sonnel  Selection  Procedures  (The  Society  for  In¬ 
dustrial  arxl  Organizational  Psychology,  Inc.,  1985), 
and  the  Standards  for  Educational  and  Psycho¬ 
logical  Testing  (American  Educational  Research 
Association,  American  Psychological  Association, 
and  National  Council  on  Measurement  in  Educa¬ 
tion,  1985). 

Structure  of  the  report 

This  report  consists  of  six  major  sections.  The 
first  section  provides  a  general  background  for  the 
occupation,  plus  hiring  and  validation  standards. 
The  second  section  documents  the  jcb  analyses 
conducted  to  support  development  of  FA.\  Acad¬ 
emy  course  50232.  The  third  section  describes  how 
the  course  was  developed,  based  on  available  job 
information  in  consultation  with  subject  matter  ex¬ 
perts  and  professional  psychologists.  Next,  the 
course,  as  it  evolved,  is  briefly  described.  The  fifth 
section  of  the  study  evaluates  the  content  validity  of 
FAA  Academy  course  50232,  as  assessed  by  a  panel 
of  subject  matter  experts.  Finally,  the  report  con¬ 
cludes  with  brief  recommendations  on  the  use  of  the 
FSS  Screen  in  personnel  selection  decisions. 

BACKGROUND 

Air  traffic  controllers 

The  Air  Traffic  Control  Specialist  (ATCS)  oc¬ 
cupation  (federal  job  series  “GS-2152”)  has  ^ree 
areas  of  specialization,  or  “options:”  flight  service 
specialists,  terminal  controllers,  and  en  route  con¬ 
trollers.  Air  traffic  control  specialists  in  the  flight 
service  option  provide  assistance  to  over  700,000 
licensed  pilots  who  fly  civilian  aircraft  in  the  United 
States,  Virgin  Islands,  and  Puerto  Rico.  Flight  ser¬ 
vice  spe(;ialists  provide  information  on  their  station’s 
service  area,  including  terrain,  preflight  and  inflight 
weather,  suggested  routes  and  altitudes  that  appear 
to  have  few  weather  problems,  indications  of  turbu¬ 
lence,  icing;  and  any  other  information  important  to 
the  safety  of  a  flight.  Specialists  enter  fUght  plan 
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data  into  the  computerized  data  base  that  is  used  to 
track  aircraft  throughout  the  national  airspace.  They 
also  utilize  the  computer  system  to  obtain  current 
and  forecast  weather  information  from  the  National 
Weather  Service  (NWS)  in  order  to  assess  weather 
conditions  over  intended  routes  of  flight  Specialists 
also  may  be  required  to  relay  air  traffic  control 
instructions,  assist  pilots  in  emergency  situations, 
provide  airport  advisory  services,  and  initiate  and 
participate  in  searches  for  missing  or  overdue  air¬ 
craft.  TTie  flight  service  system  is  presently  being 
modernized  under  the  FAA’s  National  Airspace 
System  (NAS)  plan.  Facilities  are  being  coqsoli- 
dated  in  most  regions  into  automated  flight  service 
stations  (AFSSs).  An  AFSS  serves  a  large  geo- 
grai^cal  area  and  utilizes  state-of-the-art  data  pro¬ 
cessing  and  communications  equipment  One  im¬ 
portant  innovation,  from  the  FSS  perspective,  is  the 
greater  use  of  long-distance  telephone  services  and 
other  automated  and  direct  user  devices. 

Terminal  option  air  traffic  control  specialists 
serve  at  FAA  airport  towers  and  associated  terminal 
radar  control  (TRACON)  facilities.  They  direct  air 
traffic  in  and  out  of  airports,  issue  taxi  and  takeoff 
instructions  and  air  traffic  clearances,  and  provide 
advice  based  on  their  own  observations  and  infor¬ 
mation  received  from  many  sources:  National 
Weather  Service  (NWS),  air  route  traffic  control 
centers  (ARTCC),  flight  service  stations  (FSS),  and 
others.  Terminal  controllers  assure  separation  be¬ 
tween  landing  and  departing  aircraft,  transfer  con¬ 
trol  of  aircraft  on  instrument  flight  rules  (IFR)  to  the 
en  route  controllers  at  the  ARTCCs,  and  receive 
control  of  DFR  aircraft  coming  into  the  terminal 
airspace  from  adjacent  facilities. 

En  route  option,  or  center  air  traffic  control 
specialists  at  the  ARTCCs  give  aircraft  instructions, 
air  traffic  clearances,  and  advice  regarding  flight 
conditions  during  the  en  route  portion  of  the  flight. 
En  route  controllers  assure  separation  between  air¬ 
craft  flying  along  the  Federi  airways  system  or 
operating  into  or  out  of  airports  not  served  by  a 
tenninal  facility.  Center  controllers  use  radar  or 
manual  (non-radar)  procedures  to  track  all  IFR 
aircraft  in  the  assigned  ARTCC  airspace.  They 
trarvsfer  control  of  aircraft  to  adjacent  facilities  when 
the  aircraft  enters  that  facility’s  airspace.  However, 
only  flight  service  station  controllers  will  be  dis¬ 
cussed  in  this  report. 

Hiring 

Persons  typically  enter  the  FSS  occupation  by 


one  of  two  routes.  On  one  hand,  persons  already 
employed  by  the  FA  A  may  transfer  from  other  FAA 
occupational  fields  or  programs  into  the  FSS  occu¬ 
pation.  For  example,  controllers  may  “switch  op¬ 
tions,”  that  is.  leave  the  en  route  or  terminal  control¬ 
ler  training  program  and  enter  into  training  as  a 
flight  service  specialist.  On  the  other  hand,  persons 
not  already  employed  by  the  agency  may  apply  for 
employment  in  the  occupation  via  the  competitive 
employment  process  adrninistered  by  the  Office  of 
Personnel  Management  (0PM).  Job  applicants,  in 
this  case,  must  meet  the  standard  0PM  employment 
criteria  in  order  to  join  the  FAA  as  probationary 
employees  and  enter  into  training  at  the  FAA  Acad¬ 
emy.  These  criteria  include: 

A.  Pass  the  written  ATCS  aptitude  test 
administered  by  0PM  or  the  FAA 
with  a  score  of  75.1  or  better,  AND 
have  3  years  of  general  experience  or 
4  years  of  college,  or  any  combination 
of  education  and  experience  equaling 
3  years  of  general  experience;  OR 

B.  Pass  the  written  ATCS  aptitude  test 
with  a  score  of  70  or  above  AND  4 
years  of  college  AND  have  1  year  of 
graduate  study  or  evidenceof  superior 
academic  achievement;  OR 

C.  Pass  the  written  ATCS  aptitude  test 
with  a  score  of  70  or  above  AND  have 
previous  air  traffic  controller  or  pilot 
experience  as  defined  in  the  ATCS 
job  announcements  from  0PM. 

Initial  training 

New  specialists  report  to  the  FAA  Academy  in 
Oklahoma  City,  Oklahoma,  for  initial  FSS  occupa¬ 
tional  training.  The  initial  FS5..  training  has  b^n 
conducted  on  a  pass/fail  basis  since  the  late  1970’s 
in  response  to  tte  1976  Congressional  recommen¬ 
dations  (Pickrel,  1979).  Ihe  training  program  was 
aimed  at  providing  FSS  candidates  with  a  training 
and  evaluation  curiculum  that  would  eliminate  the 
few  whose  performance  in  training  indicated  a  high 
probability  of  failure  in  field  assignments  (Convey, 
1986;  Pickrel,  1979).  This  older  program,  in  place 
from  1978  through  1988  with  some  modifications, 
was  organized  around  FSS  operational  positions. 
These  positions  were:  weather  observer,  broadcast, 
teletype,  flight  data,  preflight,  inflight,  and  emer¬ 
gency  services.  The  preflight,  inflight,  and  emer¬ 
gency  services  positions  included  functions  which. 
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if  not  properly  fulfilled,  could  have  potentially  cata¬ 
strophic  results.  The  program  was  used  to  make 
personnel  decisions;  its  validation  for  that  purpose 
is  described  by  Pickrel. 

Requirement  for  a  new  course 

The  flight  service  station  system  was  slated  for 
extensive  automation  under  the  NAS  plan  in  the 
mid-  1980’s.  The  FSS  job  was  expected  to  change,  as 
a  consequence,  necessitating  a  redesign  of  the  initial 
training  program.  On  October  9,  1985,  a  training 
proposal  for  revision  of  the  National  Flight  Service 
Station  Air  Traffic  Training  Program  was  approved 
by  the  then  Technical  Training  Division  (APT-300) 
of  the  FAA.  The  approved  training  proposal  speci¬ 
fied  that  AFSS  Model  1  equipment  (the  new  auto¬ 
mation  system  for  flight  service)  would  be  used  in 
the  training  to  accommodate  field  automation.  The 
Training  Development  Plan  (TDP)  presented  bv 
the  FAA  Academy  in  January  1986,  and  approved 
March9, 1986,autlK)rized  the  developmentof  course 
50232  as  the  National  Flight  Service  Initial  Qualifi¬ 
cations  Course.  FAA  Academy  course  50232  was 
developed  to  achieve  three  primary  objectives  (FAA 
Academy,  1989): 

(1)  To  establish  a  foundation  of 
knowledges  and  skills  required  for 
advanced  on-the-  job  training; 

(2)  To  test  the  ability  of  the  student  to 
leam  and  perform  related  job  fimc- 
tions;  and 

(3)  To  predict  a  student’s  probability  to 
succeed  to  the  full  performance  level 
as  a  flight  service  specialist. 

Legal  Framework  for  job  analysis,  test 

development  and  validation 

The  legal  firamewoik  for  the  development,  vali¬ 
dation,  aiKl  implementation  of  a  new  selection  pro¬ 
cedure  is  provided  by  the  Guidelines  and  relevant 
statutes  and  case  law.  Validity  of  a  selection  proce¬ 
dure  becomes  a  litigable  issue  if,  and  only  if,  the 
complaining  party  can  establish  a  proof  of  violation 
of  the  appropriate  civil  rights  statute,  most  com¬ 
monly  Tide  VII  of  the  Civil  Rights  Act  of  1964  [42 
use  200()e].  Violations  may  be  established  by 
showing  (a)  disparate  treatment  of  an  individui 
through  the  use  of  proscribed  criteria  (for  example, 
the  person  was  not  hired  because  of  his/her  race)  or 
(b)  that  “neutral  rules”  used  in  the  personnel  deci¬ 
sions  have  an  adverse  impact  against  a  protected 
class,  but  are  not  justified  by  “business  necessity.” 
In  the  case  of  adverse  impact,  the  employer  may 


present  an  affirmative  defense  that  the  employment 
practice  is  job-related.  In  Griggs  v.  Duke  Power  Co. 
(1971)  the  Supreme  Court  wrote  that  “.  .the  touch¬ 
stone  is  business  necessity.”  More  recently,  the 
Court  appeared  to  moderate  its  position  in  Wards 
Cove  Packing  Co.  v.  Antonio  (1989,  p.  2126),  con¬ 
cluding  that  “...the  employer  carries  the  buiden  of 
producing  evidence  of  a  business  justification  for 
his  employment  practice.”  There  are  five  important 
activities  that  serve  as  the  basis  for  justifying  a 
personnel  selection  procedure:  (1)  an  analysis  of  the 
job  (job  analysis);  (2)  the  construction  of  the  test  on 
the  basis  of  the  job  analysis  (test  development);  (3) 
the  ongoing  process  of  accumulating  evidence  about 
the  validity  of  the  test  (validation);  (4)  the  evalua¬ 
tion  of  test  fairness,  including  proposed  cutting 
scores;  and  (5)  an  assessment  of  alternative  selec¬ 
tion  procedures  that  might  produce  less  adverse 
impact  or  have  greater  validity  or  utility.  This  report, 
the  first  of  two,  reviews  the  job  analysis  and  test 
development  phases  and  begins  the  process  of  accu¬ 
mulating  validity  evidence.  The  second  report  will 
continue  the  assessment  of  validity  and  evaluate  test 
fairness  and  alternatives. 

JOB  ANALYSIS  TO  SUPPORT 
DEVELOPMENT 

OF  FAA  ACADEMY  COURSE  50232 

Job  analysis 

The  first  crucial  step  in  test  development  and 
validation  under  the  Guidelines  is  to  conduct  an 
appropriate  job  analysis.  Indeed,  the  Supreme  Court 
has  rejected  selection  procedures  that  had  adverse 
impact  for  lack  of  a  job  analysis  (see  Thompson  & 
Thompson,  1982,  for  a  review).  It  appears  that  the 
articulation  of  a  “business  related  justification”  for 
a  challenged  selection  procedure  will  require  evi¬ 
dence  of  some  form  of  a  job  analysis.  The  broad  job 
analysis  requirements  to  support  selection  test  de¬ 
velopment  are  reasonably  well  established  in  a  se¬ 
ries  of  legal  decisions  (see  Hogan  &  Quigley,  1986; 
Sparks,  1988;  and  Thompson  &  Thompson,  1982, 
for  substantive  reviews).  A  job  analysis  is  likely  to 
be  defensible  if  the  following  procedures  are  fol¬ 
lowed:  First,  critical  incidents  should  be  collected 
from  a  sample  of  incumbents  and  reviewed  by  a 
panel  of  subject  matter  experts  (SME)  to  develop  an 
initial  task  list  {Bridgeport  Guardians  v.  Police 
Department,  1977;  Davis  v.  Washington,  1972). 
Job-related  documents,  such  as  position  descrip¬ 
tions,  training  guides,  and  performance  evaluation 
forms  might  also  be  reviewed  to  develop  the  initial 
task  listing.  Second,  this  initial  task  listing  should 
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then  be  reviewed  and  revised  by  a  second,  indepen¬ 
dent  panel  of  SMEs  (incumbents  and  supervisors, 
for  example;  see  Guardians  Association  of  NYC 
Police  Department  v.  Civil  Service  Commission  of 
New  York,  1980).  Third,  a  survey  of  a  representative 
sample  of  incumbents  should  then  be  conducted  to 
collect  task-relevant  information,  such  as  frequency 
of  task  occurrence,  task  importance,  and  time  spent 
(Guardians  Association).  In  the  fourth  step,  that 
empirical  survey  data  can  then  be  used  to  cluster 
tasiu  into  related  groups,  which  panels  of  incum¬ 
bents  can  then  analyze  in  terms  of  the  knowledges, 
skills,  abilities,  and  other  characteristics  (KSAO) 
required  to  perform  those  task  clusters  (Contreras  v. 
City  of  Los  Angeles,  1981;  Guardians  Association). 
The  KSAOs  should  also  be  ranked  in  importance 
(Craig  V.  County  of  Los  Angeles,  1980)  and  the  level 
required  for  successful  job  performance  specified 
(Jones  V.  Human  Resources  Administration,  1975). . 
Finally,  the  job  analysis  should  be  written  (Jones  v. 
Human  Resources  Administration). 

Initial  job  analysis  conducted  by  Gee 
(1986) 

Identification  of  critical  or  important  job 

behaviors 

The  FAA  Manager  of  the  Air  Traffic  Systems, 
Plans,  and  Programs  Division  (ATR-100;  1986) 
requested  that  a  jobAask  analysis  be  conducted  to 
support  revision  of  the  flight  service  specialist  ini¬ 
tial  qualification  training  course.  The  Office  of 
Aviation  Medicine  (OAM)  procured  the.services  of 
a  private  contractor  to  conduct  the  job/task  analysis 
(FAA  Contract  DTFA01-86-P-0118.  1986).  The 
resulting  product  (Gee,  1986)  enumerated  the  job 
functions,  tasks,  and  task  elements  performed  by 
flight  service  specialists.  Gee  employed  a  “top- 
down”  analytic  strategy  to  identify  the  major  activi¬ 
ties  performed  on  the  job,  the  sub-activities  in¬ 
volved,  the  tasks  subordinate  to  or  encompassed  by 
those  sub-activities,  and,  as  appropriate,  identifica¬ 
tion  of  the  sub-tasks  involved  in  the  performance  of 
the  identified  tasks.  Three  incumbent  specialists 
from  the  Leesburg,  Virginia,  AFSS  prepared  an 
initial  list  of  activities,  sub-activities,  tuxl  tasks 
using  a  consensus  (Delphi)  methodology  as  the 
starting  point  for  the  analysis  in  late  1986  (Gee,  p. 
10).  Three  incumbent  specialists  at  the  DeRidder, 
Louisiana,  AFSS  then  revised  and  expanded  that 
initial  task  list.  Finally,  a  third  panel  of  three  full 
performance  level  (FPL)  specialists  at  the  Conroe, 
Texas,  AFSS  completed  a  final  revision  of  the 
acti  vity/sub-activityAask  list.  This  final  version  was 


also  reviewed  by  the  DeRidder  panel  and  by  course 
developers  at  the  FAA  Academy  (N unspecified).  In 
sum,  some  nine  FIT.  ^cialists  from  three  field 
facilities  derived  a  listing  of  the  major  functions  or 
activities  and  subordinate  tasks  for  the  flight  service 
occupation.  However,  the  contractor  did  not  collect 
frequency,  criticality,  or  difficulty  of  learning  rat¬ 
ings  for  the  enumerated  job  functions  or  tasks  from 
either  the  SME  panels  or  by  survey,  as  might  be 
recommended  on  the  basis  of  the  1980  Guardians 
Association  decision. 

Identification  of  required  knowledges,  skills, 
and  abilities  by  Convey  (1988) 

In  addition  to  describing  the  critical  or  impor¬ 
tant  work  behaviors,  a  jobAask  analysis  must  also 
identify  the  KSAOs  required  to  successfully  per¬ 
form  those  tasks  (see  the  Guardians  Association 
(1980)  and  Jones  (1975)  decisions  discussed  previ¬ 
ously).  Gee  did  not  identify  the  KSAOs  required  to 
perform  the  enumerated  job  functions  at  any  defined 
level  of  acceptable  performance.  However,  Dr.  John 
Convey,  an  OAM  contractor,  did  attempt  in  1988  to 
ascribe  the  human  attributes  required  to  perform  the 
functions  enumerated  by  Gee.  Those  linkages  ap¬ 
pear  not  to  have  been  submitted  to  systematic  re¬ 
view  by  either  subject  matter  experts  panels  for 
review  or  by  a  field  survey  of  incumbents. 

Myers  and  Manning  (1988)  job  analysis 

Identification  of  critical  or  important  work 

behaviors 

As  previously  discussed,  the  FAA  Academy  • 
concluded  that  Gee  analysis  of  the  FSS  job  was  not 
sufficient  to  support  selection  test  development  (Su¬ 
pervisor,  FSS  Revision  and  Development  Unit, 
AAC-934A,  December  1 , 1987).  Subsequently,  two 
personnel  research  psychologists  from  the  Civil 
Aeromedical  Institute’s  (CAMI)  Human  Resources 
Research  Division  (HRRD)  reviewed  available  FSS 
job  information,  training  guides,  handbooks,  and 
relevant  orders,  and  conferred  with  subject  matter 
experts,  as  well  as  observed  the  job  at  the  Bridge¬ 
port,  CT,  AFSS  to  develop  an  alternative  listing  of 
FSS  tasks.  This  task  listing  was  then  translated  into 
a  survey  form  in  order  to  collect  field  ratings  of 
frequency  of  performance  and  criticality  (Myers  & 
Manning,  1988). 

Frequency  was  defined  using  the  following 
Likert-type  scale:  1=0 nee  or  less  per  year;  2=2-4 
times  a  year;  3=1-3  times  a  month;  4=0 nee  a  week. 
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Table  1 

Results  of  FSS  job/task  analysis  ratings  (N=28) 


• 

Job  function 

■ 

C 

S 

2.1 

8.4 

Processing  new  flight  plans 

4.0 

1.5 

6.0 

Activating  flight  p^s 

3.4 

2.1 

7.1 

Processing  flight  plan  amendments/cancellations 

4.0 

1.2 

4.8 

Processing  flight  plan  closures 

3.8 

2.2 

8.4 

Processing  flight  plan  requests  for  clearance 

4.0 

2.2 

8.8 

Responding  to  pilot  requests  for  weather  data 

3.6 

1.9 

6.8 

Rqwrting  significant  aeronautical  data 

3.0 

1.8 

5.4 

Responding  to  pilot  requests  for  route  planning 

4.0 

1.5 

6.0 

Requesting  pilot  weather  reports 

3.4 

2.0 

6.8 

Procesi^ng  pilot  requests  for  airport  information 

4.0 

2.2 

8.8 

Maintaining  currency  in  weather  data 

1.2 

3.4 

4.1 

Responding  to  emergency  situations 

2.7 

2.2 

5.9 

Conducting  search  and  rescue  procedures 

1.3 

2.0 

2.6 

Processing  emergency  location  transmitter  signals 

1.2 

2.8 

3.4 

Orienting  lost  aircraft 

2.5 

1.9 

4.8 

Processing  aeronautical  data 

3.3 

1.6 

5.3 

Editing  data  base  messages 

3.5 

1.8 

6.3 

Processing  weather  data 

3.1 

1.4 

4.3 

Briefing  relieving  spedalist 

3.1 

1.8 

5.6 

Assuming  position  responsibility 

Monitoring  navigational  and  communications  equipment 

3.1 

1.8 

5.6 

Note:  F= Average  Frequency  rating;  C= Average  Criticality  rating;  S= Average  score,  where 
score=F  x  C. 


5=Once  a  day,  6=2-4  times  a  day,  7=Once  an  hour, 
%=2-4  times  an  hour,  9=More  than  4  times  an  hour. 

Criticality  of  performance  was  defined  in  a 
similar  manner  \=Very  unlikely  that  incorrect 
performance  will  eUrectfy  cause  injury  or  death; 
2=Moderate  likelihood  (f  injury  or  death;  3=High 
likelihood;  4=Will  certainly  result  in  injury  or  death. 

The  researchers  collected  data  at  2  locations: 
the  McAlester,  Oklahoma,  and  Wichita,  Kansas, 
AFSSs  from  among  the  24  AFSSs  that  had  been 
fully  commissioned  and  had  at  least  1  full  year  of 
operational  experience  at  that  time.  Myers  and  Man¬ 
ning  collected  data  from  26  incumbent  specialists 
and  2  training  managers  at  the  2  facilities.  The 
frequency  and  criticality  ratings  were  combined  to 
derive  an  average  score  for  each  job  function.  The 
results  of  this  analysis  are  presented  in  Table  1 .  Four 
tasks  had  higher  overall  scores  than  any  other  tasks, 
indicating  both  frequent  occurrence  and  the  critical¬ 


ity  of  correa  task  performance.  These  tasks  were: 
(1)  processing  new  flight  plans;  (2)  processing  pilot 
requests  for  clearance;  (3)  responding  to  pilot  re¬ 
quests  for  weather  data;  and  (4)  maintaining  cur¬ 
rency  in  weadier  data.  These  tasks  are  briefly  de¬ 
scribed  as  follows. 

Processing  new  flight  plans.  Pilots,  including 
general  aviation  and  military,  are  required  to  file  a 
flight  plan  indicating  the  planned  route  of  flight, 
departure  time,  and  estimated  arrival  time.  Pilots 
file  flight  plans  by  phone,  in  person  at  the  station,  or 
via  voice  recording.  The  specialist  then  processes 
the  new  flight  plan,  querying  the  pilot  as  needed  to 
clarify  or  complete  the  plan,  either  in  person  or  by 
calling  the  pilot  if  the  plan  was  Fil^  via  voice 
recording.  The  plan  is  reviewed  by  the  specialist  for 
completeness  and  errors,  verified  with  the  pilot,  and 
then  stored  in  the  automated  data  base. 
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Processing  pilot  requests  for  clearance.  A 
pilot  may  request  changes  or  modifications.  For 
example,  the  pilot  may  request  art  Instrument  Flight 
Rule  (IFR)  or  Special  Visual  Flight  Rule  (SVFR) 
clearance  in  order  to  obtain  air  traffic  control  ser¬ 
vices.  The  specialist  must  receive  that  pilot  request 
and  determine  the  appropriate  air  traffic  control 
facility  to  contact  The  specialist  then  contacts  the 
facility,  requests  the  clearance,  afld  receives  either 
approval  or  denial.  The  specialist  then  must  relay 
that  clearance  verbatim  to  the  requesting  pilot,  re¬ 
ceive  the  pilot’s  acknowledgement  or  rejection  of 
the  clearance,  and  make  the  appropriate  log  entries. 

Responding  to  pilot  requests  for  weather 
data.  Before  filing  a  flight  plan,  a  pilot  may  request 
weather  information  from  the  specialist.  The  spe¬ 
cialist  must  request  background  information  from 
the  pilot  in  order  to  choose  an  appropriate  type  of 
weather  briefing.  The  specialist  then  formulates  a 
plan  of  action  to  generate  the  weather  data  from  the 
automated  system,  evaluates  the  information,  inte¬ 
grates  the  data  into  a  briefing,  and  informs  the  pilot 
of  weather  data  in  a  specified  format.  The  specialist 
then  logs  the  pilot  briefing  contact. 

Maintaining  weather  data  currency.  The  spe¬ 
cialist  is  responsible  for  maintaining  an  on-going 
understanding  of  current  and  forecast  weather  con¬ 
ditions  order  to  assess  implications  of  changing 
meteorological  conditions  on  flights  within  the  as¬ 
signed  night  plan  area.  The  specialist  does  this  by 
observing  the  weather  display  queue  for  new  or 
changed  weatherdata,  reviewing  appropriate  weather 
maps  and  graphics  generated  by  the  automated 
system,  as  well  as  pilot  reports.  From  this  review  of 
available  data,  the  specialist  determines  weather 
trends  and  assesses  probable  impacts  on  the  routes 
of  flight  through  the  assigned  flight  plan  area  gener¬ 
ally,  and  on  any  VFR  flights,  specifically. 

Identification  of  required  knowledges, 

skills  and  abilities 

Manning  then  identified  skills  and  knowledges 
required  to  perform  the  smdied  Job  functions  on  the 
basis  of  available  job  documentation.  An  unspeci¬ 
fied  number  of  subject  matter  experts  from  the  FA  A 
Academy  Flight  Service  Revision  and  Develop¬ 
ment  Unit  (AAC-934A)  reviewed  those  skills  and 
knowledges  and  linkages  between  job  functions  and 
clusters  of  skills  and/or  knowledges  developed. 
TTiesc  linkages  will  be  reviewed  through  proposed 
rc-analysis  of  the  automated  FSS  job  as  consolida¬ 
tion  of  facilities  proceeds  under  the  NAS  plan. 


Summary 

Two  job/task  analyses  of  the  position  of  flight 
service  specialist  in  an  AFSS  were  conducted.  The 
first,  a  product  delivered  by  Gee  was  considered  an 
incomplete  foundation  for  the  development  of  a 
selection  (that  is,  screening)  program.  A  subsequent 
analysis,  conducted  by  Myers  and  Manning,  re¬ 
sult^  in  a  list  of  important  work  behaviors  and 
specification  of  required  KSAOs.  That  analysis 
served  as  an  important  input  to  the  next  step  in  the 
development  ofFAA  Academy  course  50232,  speci¬ 
fication  of  test  content  and  program  structure  in  the 
process  of  test  development. 

TEST  DEVELOPMENT 

Legal  framework  for  test  development 

The  second  crucial  step  is  to  develop  the  selec¬ 
tion  procedure  on  the  basis  of  the  job  analysis  with 
careful  attention  to  professional  and  technical  test 
construction.  A  selection  test  must  be  predicated 
upon  a  suitable  job  analysis  and  have  been  con¬ 
structed  with  “reasonable  competence”  (Guardians 
Association,  1980).  For  example,  in  Guardians  As¬ 
sociation  (p.  96),  the  selection  test  was  criticized  for 
haphazard  process  of  writing  test  items  by  “ama¬ 
teurs  in  the  art  of  test  construction”  that  did  not  have 
access  to  the  job  analysis  materials  during  test 
construction.  In  contrast,  the  employer  in  Cuesta  v. 
New  York  Office  of  Court  Administration  (1987) 
adhered  to  a  careful  test  constmction  process,  and 
was  guided  by  the  results  of  the  job  analysis  under 
the  supervision  of  a  professional  test  development 
firm.  The  employer  and  contractor  were  careful  to 
assess  the  important  aspects  of  the  job  for  which 
measurement  was  appropriate  and  feasible,  rather 
than  measuring  all  components  of  the  job  in  their 
exact  proportion  regardless  of  significance  (Cuesta 
v.New  York  Office  of  Court  Administration,  1 987 ,  p. 
46,603).  The  centraJ  lesson  of  the  Guardians  Asso¬ 
ciation  and  Cuesta  cases  is  that  the  selection  test 
must  be  competently  constructed  and  its  content 
solidly  anchored  in  the  job  analysis  in  order  to 
survive  the  rigors  of  legal  scrutiny  if  adverse  impact 
is  established. 

This  framework  suggests  two  central  issues  that 
should  be  addressed  in  order  to  establish  the  linkage 
between  the  job  analyses,  as  conducted  for  the  flight 
service  specialist  occupation  and  FAA  Academy 
course  5(X232.  The  first  issue  is,  “Were  the  results  of 
the  job  analysis  used  to  specify  the  content  of  the 
FSS  Screen?”  This  section  of  the  repoh  will  exam- 


6 


ine  the  available  documentation  linking  job  analysis 
to  program  ^jecifications.  The  second  issue  is,  “Were 
the  selected  tasks  appropriately  operationalized  in 
the  graded  instruments  from  which  scores  will  be 
computed  as  the  basis  of  future  selection  deci¬ 
sions?”  The  second  issue  will  be  addressed  in 
another  section  of  this  report. 

Specification  of  test  (screen)  content  from 
the  job  analysis 

The  next  question  is:  Were  the  results  of  the 
Myers  and  Manning  analysis  used  to  frame  the 
content  of  FAA  Academy  course  50232?  The 
products  of  the  Myers  and  Manning  job  analysis 
were  reviewed  by  subject  matter  experts  at  the  FAA 
Academy  in  the  process  of  designing  course  50232. 
Initial  decisions  about  course  content  made  in  Janu¬ 
ary  1988,  reflect  the  results  of  the  Myers  and  Man¬ 
ning  analysis,  according  to  notes  taken  and  distrib¬ 
uted  by  the  Supervisor  of  the  Flight  Service  Revi¬ 
sion  and  Development  Unit  (AAC-934A): 

"The  following  decisions  were  made  and  agreed 
upon  by  the  participants: 

“The  screen  will  consist  of  four  primary 
areas:  weather  analysis;  flight  plan  process¬ 
ing,  which  will  incorporate  Model  1  usage; 
emergency  services;  and  the  last  item  is 
what  we  called  prioritizing  requests. 

“The  weather  analysis  will  consist  of  a 
group  of  charts  and  alpha^umeric  data  that 
the  student  will  have  to  match  together.  This 
will  test  his/her  ability  to  analyze  weather 
patterns.  It  will  also  contain  a  pilot  briefing 
that  will  demonstrate  the  student’s  ability  to 
present  the  weather  in  a  summarized  form. 

“The  flight  plan  processing  will  demon¬ 
strate  the  student’s  ability  [to]  utilize  the 
Model  1  equipment. 

“The  emergency  services  will  be  the  V OR 
[Variable  Omni-directional  Range]  and  ADF 
[Automatic  EHrection  Finding]  orientations 
from  the  present  course. 

“The  last  item,  prioritizing  requests,  will 
involve  multiple  simultaneous  radio  con¬ 
tacts  where  the  student  must  identify  the 
highest  priority  requests  and  provide  the 
service  in  the  proper  order.” 


On  one  hand,  the  weather  analysis  and  flight 
plan  processing  components  clearly  flow  directly 
from  the  results  of  the  job/task  analysis.  On  the  other 
hand,  the  inclusion  of  emergency  services,  in  the 
form  of  orienting  lost  aircraft,  in  the  proposed  Screen 
is  the  product  of  SME  review  of  the  job/task  analy¬ 
sis.  The  fourth  component,  prioritizing  requests, 
appears  not  to  have  been  operationalized  in  the  FSS 
Screen. 

Discussions  with  the  CAMI  researcher  present 
at  the  January  11,  1988,  meeting  suggest  that  the 
discussants  believed  that  the  overall  rating  for  the 
“orienting  lost  aircraft”  task  did  not  truly  reflect  its 
importance  to  the  occupation  because,  while  the 
fr^uency  of  the  task  was  low,  its  criticality  was 
very  high.  Discussants  at  this  meeting  inclwled  six 
subject  matter  experts  (full  performance  level  spe¬ 
cialists)  serving  at  the  FAA  Academy  as  course 
developers.  They  concluded  that  this  task  (orienting 
aircraft)  is  an  emergency  situation  in  which  it  is 
“very  likely  that  incorrect  performance  will  directly 
cause  injury  or  death.”  This  assessment  reflected 
the  criticality  ratings  provided  by  incumbents  in  the 
Myers  and  Manning  job  analysis. 

Participants  at  the  January  11,  1988,  meeting 
justified  dropping  the  “processing  flight  plan  re¬ 
quests  for  clearance”  task  from  the  proposed  FSS 
Screen  because  specialists  simply  relay,  verbatim, 
clearances  received  from  termini  or  en  route  con¬ 
trollers.  Moreover,  discussants  concluded  that  the 
relevant  skills  and  knowledges  were  taught  and 
tested  in  the  flight  plan  processing  component  of  the 
proposed  FSS  Screen.  Therefore,  although  the  “pro¬ 
cessing  flight  plan  requests”  task  was  rated  in  the  job 
analysis  as  “^quent”  and  “somewhat  critical,”  it 
was  perceived  by  the  Academy  SME  panel  as  being 
easy  to  train  and  as  not  requiring  any  skills  and 
knowledges  not  already  assessed  in  the  proposed 
course. 

Finally,  another  subject  matter  expert  meeting 
was  conducted  on  February  1 1 , 1988,  to  discuss  the 
weather  analysis  portion  of  the  proposed  course. 
Again,  from  notes  taken  and  distributed  by  the 
Supervisor  of  the  FSS  Revision  and  Development 
Unit  (AAC-934A)  for  that  meeting,  participants 
determined  that  two  primary  weather-related  skills 
were  implied  by  the  job/task  analysis: 

“The  first  skill  is  to  maintain  currency  on 
weather  producing  patterns  and  the  second  is 
the  ability  to  respond  to  a  pilot’s  request  for 
weather  data.” 
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The  participants,  agreed  at  this  meeting  that 
these  two  tasks,  identified  in  the  Myers  and  Man¬ 
ning  job  analysis,  would  be  operationalized  in  two 
separate  laboratory  simulations.  Maintaining  cur¬ 
rency  was  operationalized  as  a  laboratory  exercise 
requiring  the  specialist  to  observe  to  determine 
weather  trends  from  review  of  appropriate  weather 
maps  and  graphics.  Responding  to  pilot  requests  for 
weather  data  was  operationalized  as  a  separate  labo¬ 
ratory  exercise  to  evaluate  current  information  in 
light  of  trends  and  to  inform  the  pilot  of  that  assess¬ 
ment  using  appropriate  phraseology. 

Development  of  the  course 

Development  of  FAA  Academy  course  50232 
is  more  Sally  described  in  the  formal  course  report 
prepared  by  the  FAA  Academy  Instructional  Sys¬ 
tems  staff  (FSS  Instructional  Systems  Unit,  AAC- 
934A,  December  1990).  The  NWS  unit  assigned  to 
the  FAA  Academy  developed  the  weather  analysis 
tests  and  laboratory  simulations  based  on  the  FAA 
specifications.  IThe  remainder  of  the  course  was 
developed  by  FPL  specialists  assigned  to  the  FAA 
Academy  during  1988.  A  doctorate-level  Instruc¬ 
tional  Systems  Developer  joined  the  development 
staff  mid-year  1989  to  provide  professional  guid¬ 
ance  and  oversight  to  the  development  of  instruc¬ 
tional,  testing,  and  laboratory  materials  for  course 
50232.  In  addition,  personnel  research  psycholo¬ 
gists  from  CAMI  provided  consultative  services 
throughout  the  development  of  course  materials  for 
the  FSS  Initial  Qualifications  Course.  FAA  Acad¬ 
emy  course  50232  represents  an  evolutibn  of  previ¬ 
ous  FSS  training  programs,  rather  than  a  radical 
departore,  in  the  same  way  that  field  automation  and 
consolidation  of  FSSs  represents  an  evolution,  rather 
than  revolution,  of  the  air  traffic  system. 

Summary 

It  appears  that  the  Myers  and  Manning  (1988) 
analysis  of  the  FSS  job  in  AFSSs  was  used  to  specify 
the  content  of  the  proposed  FAA  Academy  course 
50232.  On  one  hand,  the  flight  plan  processing  and 
weather  analysis  elements  of  FAA  Academy  course 
50232  can  be  traced  directly  to  the  Myers  and 
Manning  analysis.  On  the  other  hand,  the  inclusion 
of  emergency  services  in  the  form  of  “orienting  lost 
aircraft”  as  part  of  course  50232  is  the  direct  result 
of  decisions  made  by  a  panel  of  subject  matter 
experts  upon  review  of  the  jobAask  analysis  results. 
The  development  of  course  50232  was  accomplished 
through  the  collaboration  of  SMEs  and  professional 
psychologists  and  an  instructional  systems  designer. 


FAA  Academy  course  50232  was  developed  with 
reasonable  competence  and  care  in  accordance  with 
existing  professional  and  technical  guidelines,  stan¬ 
dards,  and  practices. 

DESCRIPTION  OF  FAA  ACADEMY 
COURSE  50232 

The  overall  course  sped  fications  for  FAA  Acad¬ 
emy  course  50232  resulted  from  this  series  of  meet¬ 
ings  between  Academy  subject  matterexpeits/course 
developers  and  CAMI  researchers.  The  develop¬ 
ment  of  actual  course  materials  is  documented  in  the 
final  course  report  from  the  FSS  Revision  and  De¬ 
velopment  Unit  to  the  Air  Traffic  Branch  of  the  FAA 
Academy  (Flight  Service  Instructional  Systems  Unit, 
A  AC-934 A,  November  1990).  The  general  outlines 
of  the  final  course  are  presented  below. 

Four  major  occupational  topics  are  covered  by 
the  course  in  both  classroom  and  applied  laboratory 
settings:  (a)  general  academics;  (b)  flight  plan 
processing  using  Model  1  AFSS  equipment;  (c) 
weather  analysis;  and  (d)  aircraft  orientation.  TTie 
general  academics  instructional  block  is  designed  to 
provide  basic  aviation  information  needed  for  job 
performance  and  knowledgeable  communication 
with  pilots.  Students  also  memorize  the  relevant 
characteristics  of  the  FAA  Academy  local  flight 
plan  area,  the  geographical  region  for  which  the 
specialist  is  responsible  in  the  Icboratory  simula¬ 
tions  later  in  the  course.  Knowledge  gained  in  this 
block  of  instruction  is  tested  in  the  General  Academ¬ 
ics  Block  Test 

The  flight  plan  processing  components  of  course 
50232  introduce  the  student  to  the  Flight  Service 
Automation  System  (FSAS),  Model  1  equipment, 
procedures,  terminology  and  briefing  functions. 
FSAS  briefing  functions  enable  the  specialist  to  call 
up  various  pieces  of  information  from  the  computer 
data  base  of  flight  plans  and  weather  data. 
Knowledges  gained  in  this  block  of  instruction  are 
tested  in  the  General  Academics  Block  Test;  both 
knowledges  and  skills  are  tested  in  the  flight  plan 
processing  laboratory  exercises. 

The  Weather  Analysis  instructional  block,  de¬ 
veloped  and  administered  by  the  NWS  in  coopera¬ 
tion  with  the  FAA  Academy,  provides  the  basic 
meteorological  information  required  to  effectively 
evaluate  the  probable  impact  of  changing  weather 
conditions  on  intended  routes  of  flight  Students,  in 
the  course  of  this  instructional  block,  leant  to  recog- 
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nize  the  causes  and  effects  of  specified  weather 
patterns.  They  also  learn  to  extract  relevant  iitforma- 
tion  from  the  weather  data  base,  using  the  FSAS 
Model  1  equipment  in  order  to  determine  whether 
weather  conditions  could  pose  a  potential  danger  to 
proposed  flight  plans.  The  Weather  Analysis  Block 
Test  is  oriented  towards  testing  of  knowledges  gained 
through  academic  instruction;  the  Weather  Analysis 
Controller  SkiUs  Tt^t  (CST),  on  the  other  hand,  is 
intended  to  be  an  integrated  test  of  knowledges  and 
skills.  The  applied  laboratory  exercises  allow  the 
students’  kiK}wledges  and  skills  to  be  systematically 
evaluated  in  a  “hands-on”  simulation  of  gathering 
information,  integrating  it,  and  assessing  potential 
impact  on  air  traffic. 

Finally,  the  Aircraft  Orientation  component  of 
the  FSS  Screen  teaches  the  student  two  distinct 
methods  for  providing  aircraft  orientation  services. 
The  student  then  applies  those  knowledges  in  labo¬ 
ratory  simulations  of  lost  aircraft  situations  and  also 
in  a  final  CST.  Both  the  laboratory  exercises  and  the 
CST  allow  the  students’  knowledges  and  skills  to  be 
systematically  evaluated. 

The  first  class  entered  course  50232  in  March 
1989,  on  a  non-pass/fail  (that  is,  all  pass)  basis.  The 
program  has  continued  on  a  non-pass/fail  basis 
pending  stabilization  of  course  content  and  stan¬ 
dardization  of  course  administration  (Broach,  1989). 
In  June  1990,  the  Manager  of  the  AirTraffic  Branch 
(AAC-930)  determined  that  course  50232  was  suf¬ 
ficiently  stabilized  in  content  and  standardized  in 
administration  to  warrant  validation  for  use  in  mak¬ 
ing  personnel  decisions. 

TEST  VALIDATION 

Legal  framework  for  test  validation 

The  third  important  step  is  to  accumulate  evi¬ 
dence  concerning  the  validity  of  the  newly-devel¬ 
oped  selection  test  using  an  appropriate  data  collec¬ 
tion  strategy.  Validity  refers  to  “the  appropriate¬ 
ness,  meaningfulness,  and  usefulness  of  the  specific 
inferences  made  from  test  scores”  (American  Edu¬ 
cational  Research  Association,  and  others,  1985,  p. 
9).  Traditionally,  the  data  collection  strategies  em¬ 
ployed  to  accumulate  validity  evidence  have  been 
grouped  into  three  categories:  (a)  content-related; 
(b)  criterion-related;  and  (c)  construct-oriented. 
Content-related  evidence  for  the  validity  of  a  selec¬ 
tion  procedure  demonstrates  the  degree  to  which  the 
sample  of  items,  tasks,  or  questions  on  a  test  are 
representative  of  some  defined  universe  or  domain 


of  content.  For  example,  systematic  ot  servations  of 
job  behavior  may  be  combined  with  expert  judg¬ 
ment  to  construct  a  sample  of  critical  or  important 
job  behaviors  that  can  be  auninistered  under  stan¬ 
dardized  conditions. 

The  first  task  for  developers  is  the  adequate 
specification  of  the  domain  that  the  proposed  test  is 
intended  to  represent,  given  the  probable  use  of  the 
test.  In  other  words,  a  job  analysis  must  be  con¬ 
ducted  and  used  to  derive  the  specifications  for  the 
test,  as  discussed  above.  Anotlier  important  task  for 
developers  is  the  determination  of  the  degree  to 
which  item  or  work  sample  formats  ?nd  response 
properties  are  representative  of  the  job  uomain  to  be 
sampled.  That  is,  an  appropriate  testing  strategy 
must  be  employed.  Criterion-related  evidence,  on 
the  other  hand,  demonstrates  Jiat  the  test  scores  are 
systematically  related  to  one  or  more  Job  perfor¬ 
mance  criteria  or  outcomes.  The  intrinsic  value  of  a 
criterion-related  study  as  evidence  for  the  validity  of 
a  selection  procedure  depends  on  the  relevance  of 
the  job  performance  criterion  or  organizational  out¬ 
come  that  is  used  (American  Educational  Research 
Association,  et.  al.,  1985,  p.  1 1).  Finally,  construct- 
oriented  validation  focuses  primarily  on  the  test 
score  as  a  measure  of  ^  psychological  characteristic 
thought  to  underlie  successful  performance  of  criti¬ 
cal  or  important  work  behaviors.  The  choice  of  a 
validation  strategy  (content-,  criterion-or  construct- 
related)  is  a  function  of  professional  judgment  in 
light  of  the  intended  inferences  from  test  scores. 

Strategy^ 

Choice  of  validation  strategy 

The  choice  of  validation  strategy,  in  this  case, 
was  dictated  by  two  constraints.  First,  a  predictive  or 
concurrent  criterion-related  validation  strategy  was 
not  technically  feasible  as  defined  by  the  Uniform 
Guidelines  (section  14.B.[1]).  Specifically,  reliable 
measures  of  field  job  performance  with  appropriate 
statistical  properties  are  not  currently  available  for 
the  flight  service  occupation.  Moreover,  there  is 
little  attrition  from  field  training  among  the  students 
who  passed  through  the  program  since  March  1989. 
Second,  the  Uniform  Guidelines  (section  14. C.  [7]) 
provides  that 

“Where  a  measure  of  success  in  a  training 
program  is  used  as  a  selection  procedure ...  the 
use  should  be  justified  on  the  relationship  be¬ 
tween  the  content  of  the  training  program  and 
the  content  of  the  job.” 
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Success  —  or  lack  of  —  in  course  50232  will  be 
used  as  the  basis  for  retention  or  separation  from  the 
flight  service  occupation  as  described  previously. 
Therefore,  a  content-oriented  validation  strategy 
was  selected  for  this  study. 

Technical  standards 

To  establish  content  validity  under  the  Uniform 
Guidelines  (Section  14.C.[4]), 

“For  any  selection  procedure  measuring  a 
knowledge,  skill,  or  ability  the  user  should 
show  that  (a)  the  selection  procedure  mea¬ 
sures  and  is  a  representative  sample  of  that 
knowledge,  skill,  or  ability;  and  (b)  that 
knowledge,  skill,  or  ability  is  used  in  and  is 
a  necessary  pre-requisite  to  performance  of 
critical  or  important  work  behaviorfs).” 

In  this  case,  what  is  required  is  a  showing  that 
the  components  of  the  selection  procedure;  e.g.,  the 
graded  tests  and  laboratory  exercises,  sample  the 
knowledges  and  skills  required  to  perform  the  im¬ 
portant  job  functions  identified  in  the  various  job/ 
task  analyses.  The  linkage  between  knowledges  and 
skills  and  the  selection  procedure  is  most  easily 
established  when  the  results  of  the  job  analysis 
clearly  served  as  the  blueprint  for  the  test  and  when 
the  process  of  test  development  is  well  documented 
{Chance  v.  Board  of  Examiners,  1971;  Common¬ 
wealth  of  Pennsylvania  v.  O’Neill,  1979;  Cuesta  v. 
New  York  Office  of  Court  Administration,  1987). 
The  instructional  systems  design  process  used  by 
the  FAA  Academy  and  consultative  services  pro¬ 
vided  by  personnel  research  psychologists  from  the 
HRRD  provided  some  assurance  that  instructional 
and  test  content  would  be  anchored  in  the  available 
job  information  and  management-defined  course 
objectives.  However,  a  review  of  the  graded  compo¬ 
nents  by  subject  matter  experts  from  the  field  was 
chosen  as  a  final  quality  control  check  on  course 
content,  as  represented  by  the  graded  instniments, 
vis-a-vis  the  available  knowledge  and  task  informa¬ 
tion  for  the  job  functions  that  are  tested  in  course 
50232. 

Method 

Subject  matter  expert  (SME)  panel 
A  panel  of  1 6  full  performance  level  automated 
flight  service  specialists  was  assembled  in  Okla¬ 
homa  City  for  the  week  of  September  10-14, 1990. 
Six  of  the  specialists  were  currently  assigned  as 
Assistant  Managers  forTraining  in  the  home  facili¬ 


ties;  the  other  10  specialists  were  non-supervisory, 
active  duty  personnel  performing  the  day-to-day  job 
of  a  specialist.  The  mean  age  for  the  group  was  42, 
and  ranged  from  29  to  53.  Overall,  the  group  had 
been  at  the  full  performance  level  (FPL)  for  an 
average  of  12  years.  The  least-experienced  special¬ 
ist  had  1  year  as  an  FPL  controller,  the  most 
experienced  had  28  years  in  service  as  an  FPL 
specialist.  There  were  three  women  in  the  panel. 
However,  it  must  be  noted  that  none  had  more  than 
three  years  of  experience  in  AFSSs,  in  view  of  the 
recency  of  consolidation  and  automation.  More¬ 
over,  the  panel  size  was  limited  by  practical  con¬ 
straints  of  the  availability  of  personnel  and  funds  for 
travel.  The  majority  of  the  panel  declined  to  provide 
race/ethnic  data  (13  of  16).  Five  of  the  controllers 
had  a  4-year  college  degree;  10  reported  at  least 
some  college  education,  while  1  reported  a  high 
school  degree  as  the  highest  degree  of  education 
completed. 

Procedure 

Raters  were  provided  with  a  rating  sheet  for 
each  graded  component  of  the  FSS  Screen.  The 
rating  sheets  listed  the  knowledges  and  tasks  asso¬ 
ciated  with  each  major  instructional  block,  as  iden¬ 
tified  in  the  Myers  and  Manning  job  analysis,  and 
particularly.  Manning’s  analysis  of  knowledges  re¬ 
quired  to  perform  AFSS  job  functions^.  The  rating 
sheet  provided  formaking  a“yes’’or“no’’ judgment 
as  to  whether  the  knowledge  was  represented  by  at 
least  one  item  of  the  relevant  block  test.  Raters 
evaluated  whether  tasks  aod.  knowledges  were  rep¬ 
resented  by  laboratory  exercises  and  CST  on  the 
premise  that  these  instruments  represented  integra¬ 
tions  of  knowledge  and  behavior.  Raters  read  through 
the  academic,  multiple-choice  type  examinations 
(the  Block  Tests  and  CST)  or  worked  through  the 
problem  scenarios  in  the  AFSS  laboratory.  Due  to 
time  constraints,  only  the  final  graded  problem  for 
each  block  was  examined.  The  results  of  this  study 
are  therefore  based  on  a  sample  of  the  laboratory 
problems;  the  adequacy  of  the  laboratory  simula¬ 
tions  can  only  be  infen^  from  this  sample,  rather 
than  unequivocally  demonstrated. 

Analyses 

First,  the  overall  proportion  of  raters  assigning 
a  “yes”  to  each  task  or  knowledge  element  for  a 
graded  instrument  was  computed.  For  example,  16 
of  1 6.  or  100%  of  the  raters  indicated  the  “principles 
of  aerodynamics”  were  tested  in  the  General  Aca¬ 
demics  Block  Test  (Table  2).  Second,  the  proportion 
of  tasks  or  knowledge  elements  assigned  a  “yes” 
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Table  2 

Proportion  of  raters  (N=16)  indicating  that  the  General  Knowledge  elements 
were  represented  in  the  General  Academics  Block  Test 


General  Knowledge  element  BT 


Principles  of  aerodynamics  100.0% 

Aircraft  types  37,5 

Aircraft  performance  characteristics  100.0 

Navigational  aids  (Radio  beacons,  VORs,  etc.)  50.0 

Aeronautical  Charts  and  symbology  93.8 

Federal  airway  system  100.0 

Regulations  governing  VFR/IFR  operations  (FAR,  etc.)  100.0 

Phonetic  alphabet/numbers  in  communications  87.5 

Aircraft  identifiers  and  call  signs  6.3 

Facility  identifiers  87.5 

Phraseology  93.8 

Priority  of  duties  68.8 

Local  flight  plan  area  topographic  features/landmarks  87.5 

Local  flight  plan  area  navigation  aids  87.5 

Local  flight  plan  area  airways,  airports,  and  mileages  87.5 

Local  flight  plan  area  restrict^  areas/airspaces  87.5 

Local  flight  plan  area  communication  frequencies  81.3 


Overall  %  general  knowledge  elements  represented  80.9 


Note:  BT= Block  Test 


across  elements  was  computed  for  each  rater.  For 
example,  rater  1  may  have  indicated  that  14  of  the 
17,  or  88%,  of  the  “general  knowledge  elements” 
identified  in  the  job/task  analysis  were  covered  by 
the  General  Academics  Block  Test  Rater  2  may 
have  indicated,  in  contrast,  that  all  (100%)  of  these 
elements  were  tested  in  general  academics,  and  so 
on  for  each  of  the  1 6  raters.  These  percentages  were 
then  averaged  across  raters  (percent  “yes”  across 
elements  and  raters)  to  provide  an  indication  of  how 
well  the  test  sampled  ftom  the  task  and  knowledge 
domain  associated  with  each  job  fuiKtion.  For  ex¬ 
ample,  on  the  average,  the  rating  panel  indicated  that 
about  81%,  or  14  of  the  17,  of  the  “general  knowl¬ 
edge  elements”  were  represented  or  tested  in  the 
General  Academics  Block  Test  (Table  2).  A  crite¬ 
rion  of  70%  coverage  was  selected  as  the  mitumally 
acceptable  sampling.  This  somewhat  arbitrary  crite¬ 
rion  was  select^  because  it  was  consistent  with  the 
common  standard  used  by  the  FAA  Academy  in 
assessing  student  competency  and  it  assured  that  the 
task  domain  was  reasonably  well  sampled.  Panel 


members  did  not  evaluate  the  appropriateness  of  a 
testing  format  for  a  particular  taiowledge  or  skill; 
nor  were  panel  members  free  to  add  or  delete 
knowledges  or  skills. 

Results!^ 

General  Academics 

Overall,  raters  indicated  that  about  80%  of  the 
listed  general  knowledge  elements  from  the  JobAask 
analysis  were  represented  in  the  General  Academics 
Block  Test  (Table  2).  Three  specific  topics,  how¬ 
ever,  were  not  weU  covered  in  the  test:  (1)  aircraft 
types;  (2)  navigational  aids;  and  (3)  aircraft  identi¬ 
fiers  and  call  signs.  The  general  knowledges  appear 
to  be  adequately  sampled  for  the  purpose  of  estab¬ 
lishing  the  content  vtdidity  of  the  test 

Flight  Plan  Processing 

Overall,  the  SME  panel  indicated  that  about 
69%  of  the  knowledges  associated  with  flight  plan 
processing  in  the  available  JobAask  analyses  were 
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Table  3 

Proportion  of  raters  (N=  16)  indicating  that  the  Flight  Plan  Processing  elements 
were  represented  in  the  Flight  Plan  Processing  block  test  and/or  laboratory  problem 


Flight  Plan  Processing  element 

BT 

Lab 

Knowledges 

FSAS/FSDPS  configuration 

93.8 

6.3 

CRT  display  operation 

62.5 

93.8 

Main  keyboard  layout  and  operation 

81.3 

87.5 

Keyboard  function  keys 

100.0 

100.0 

Screen  editing  procedures 

93.8 

75.0 

Flight  plan  mask  display  procedures 

43.8 

93.8 

Data  entry  procedures 

93.8 

100.0 

Flight  plan  data  elements 

100.0 

100.0 

Keywords  for  filing  plans 

25.0 

100.0 

Flight  plan  display  procedure 

100.0 

43.8 

Overall  %  Flight  Plan  Processing  knowledges  represented 

69.4 

80.0 

Tasks 

Receive  request  to  file  flight  plan  via  phone  or  walk-in 

100.0 

File  flight  plan 

100.0 

Query  pilot  about  flight  plan 

100.0 

Review  flight  plan  for  completeness  and/or  errors 

100.0 

Verify  flight  plan  with  pilot 

62.5 

Store  flight  plan  in  data  base 

100.0 

Overall  %  Flight  Plan  Processing  tasks  represented 

93.8 

Note:  BT= Block  Test;  Lab = Graded  laboratory  exercise 


represented  by  the  flight  plan  processing  portion  of 
the  General  Academics  Block  Test  (Table  3).  Three 
elements  were  perceived  as  riot  well  covered  by  the 
panel:  (1)  CRT  display  operation;  (2)  flight  plan 
display  procedures;  and  (3)  keywords  for  filing 
fli^t  plans.  The  flight  plan  processing  portion  of 
the  General  Academics  Block  Test  appears  to  be 
only  marginally  acceptable  for  the  purpose  of  estab¬ 
lishing  the  content  v^idity  of  the  test.  However,  the 
identified  weaknesses  can  be  easily  ameliorated  by 
expeditious  item  writing,  try-out,  and  incorporation 
into  the  block  test  The  flight  plan  processing  labo¬ 
ratory  problem,  in  contrast  appears  to  sample  the 


relevant  knowledge  and  task  domain  adequately, 
with  80%  of  the  identified  knowledge  and  94%  of 
the  listed  task  elements  represented.  Overall,  it 
appears  that  the  flight  plan  processing  laboratory 
problem  adequately  sampled  the  knowledge  and 
task  domains. 

Aircraft  orientation 

Overall,  the  SME  panel  indicated  that  99%  of 
the  knowledges  associated  with  aircraft  orientation 
in  the  presented  jobAask  information  were  repre¬ 
sented  in  the  CST  (Table  4).  About  74%  of  the 
delineated  tasks  were  also  represented  in  the  CST. 
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Table  4 

Proportion  of  raters  (N-16)  indicating  that  the  Aircraft  Orientation  elemerus 
were  represented  in  the  Aircraft  Orientation  written  test  and/or  laboratory  problem 


Aircraft  Orientation  element 

CST 

ADF 

VOR 

Knowledges 

VOR  navigation  terminology 

100.0 

0 

100.0 

Plotting  procedures 

100.0 

100.0 

100.0 

Coordination  procedures 

100.0 

100.0 

100.0 

ADF  navigation  terminology 

100.0 

100.0 

0 

Airborne  receiver  component  functions 

100.0 

93.8 

93.8 

Use  of  orientation  aids 

100.0 

87.5 

100.0 

Radio  procedures  and  phraseology 

100.0 

100.0 

100.0 

Information  needed  from  pilot  to  orient  aircraft 

100.0 

100.0 

100.0 

Procedures  for  determining  heading  and  altitude 

93.8 

93.8 

100.0 

Procedures  for  establishing  aircraft  on  inbound  heading 

100.0 

100.0 

100.0 

Overall  %  Aircraft  Orientation  knowledges  covered 

99.4 

97.2 

89.4 

Tasks 

Choose  method  of  aircraft  orientation  (VOR  or  ADF) 

100.0 

93.8 

100.0 

Conduct  aircraft  orientation  services 

100.0 

100.0 

100.0 

Determine  if  pilot  knows  how  to  operate  equipment 

25.0 

31.3 

31.3 

Instruct  pilot  on  aircraft  equipment  operation  if  needed 

100.0 

87.5 

93.8 

Perform  required  mathematical  calculations 

37.5 

25.0 

0 

Enlist  assistance  of  any  other  aircraft  in  area 

0 

0 

0 

Receive  pilot  request  for  guidance  to  airport 

100.0 

100.0 

100.0 

Issue  course  instructions  and  advisories  to  pilot 

100.0 

100.0 

93.8 

Terminate  aircraft  orientation 

100.0 

100.0 

100.0 

Complete  orientation  record 

75.0 

93.8 

100.0 

Overall  %  Aircraft  Orientation  tasks  covered 

73.8 

73.1 

71.9 

Note:  CST= Controller  Skills  Test;  ADF= Automatic  Direction  Finding  graded  laboratory 
exercise;  VOR= Variable  Omni  Range  graded  laboratory  exercise 


However,  in  the  view  of  the  panel,  there  were  three 
areas  that  were  not  well  sampled,  as  indicated  by 
element  proportions  ofless  than  70%:  (1)  determin¬ 
ing  if  tlK  pilot  (of  the  lost  aircraft)  knew  how  to 
operate  the  equipment;  (2)  performing  required 
mathematical  calculations  to  obtain  time  and  dis¬ 
tance;  and  (3)  enlisting  the  assistance  of  other  air¬ 
craft.  The  first  and  third  areas  may  be  corrected  by 


future  revisions  and  development.  The  low  rating  of 
mathematical  calculations  was  considered  to  be 
suspect  because  of  the  wording  of  the  task  state¬ 
ment;  time  and  distance  computations  are  not  per¬ 
formed  in  the  course  of  ADF  or  VOR  aircraft  orien¬ 
tations.  The  aircraft  orientation  laboratory  problems 
for  both  methods  (ADF  and  VOR)  presented  the 
same  pattern  of  general  and  specific  results.  There- 
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Table  5 

Proportion  of  raters  (N=16)  indicating  that  the  Weather  Analysis  elements 
were  represented  in  the  weather  written  tests  and/or  laboratory  problems 


Weather  Analysis  element 

BT 

CST 

BIG 

RTE 

Knowledges 

Weather  equipment  parameter  selection/entry 

6.3 

0 

50.0 

50.0 

Detailed  weaker  data  display  procedure 

12.5 

5.9 

25.0 

68.8 

Weather  message  order 

87.5 

41.2 

0 

50.0 

Weather  reports/displays 

lOO.l, 

94.1 

75.0 

100.0 

Types  of  forecasts/advisories 

100.0 

94.1 

100.0 

100.0 

Ty^s  and  characteristics  of  weather  systems 

100.0 

94.1 

100.0 

100.0 

Weather  charts  and  symbols 

100.0 

94.1 

100.0 

100.0 

Overall  %  Weather  Analysis  knowledges 

72.3 

60.7 

50.8 

71.1 

represented 

Tasks 

Observe  display  queue  for  new/changed  weather  data 

0 

0 

37.5 

Detect  updates  in  weather 

17.6 

12.5 

100.0 

Review  weather  maps  and/or  color  weather  graphics 

94.1 

100.0 

100.0 

Review  continuously  all  weather  updates  and  reports 

25.0 

6.3 

81.3 

Determine  weather  trends 

50.0 

87.5 

100.0 

Detenuine  impact  of  weather  on  routes 

88.2 

100.0 

100.0 

Determine  impact  of  weather  on  VFR  flight 

17.6 

100.0 

100.0 

Convey  descriptive  summary  of  current  en  route  weather 

0 

87.5 

62.5 

Assist  pilot  in  selecting  alternate  routes  or  destinations 

5.9 

0 

100.0 

Overall  %  Weather  Analysis  tasks  represented 

34.7 

54.9 

86.6 

Note:  BT=BIock  Test;  CST= Controller  Skills  Test;  BIG="Big  Picture"  Weather  Analysis 
graded  laboratory  exercise;  RTE= "Route  Weather"  Weather  Analysis  graded  lalraratocy 
exercise 


fore,  the  aircraft  orientation  components  of  the  FSS 
Screen  adequately  sampled  the  relevant  knowledge 
and  skill  domains. 

Weather  analysis 

Overall,  the  weather  analysis  graded  compo¬ 
nents  of  the  FSS  Screen  are  the  weakest  (Table  5). 
The  Block  Test  appears  to  adequately  sample  the 
associated  knowledge  domain  (72%  coverage  over¬ 
all).  However,  the  Weather  Analysis  CST  did  not 
adequately  represent  either  knowledges  (6 1  %  over¬ 
all)  or  task  elements  (35%  overall).  Three  1^  wledges 
were  not  adequately  sampled  in  the  rating  panel’s 


view:  ( 1 )  weathermessage  order,  (2)  detailed  weather 
data  display  procedures;  and  (3)  equipment  param¬ 
eter  selection/entry.  Just  two  of  nine  task  elements 
were  perceived  by  the  panel  as  being  adequately 
represented  in  the  CST  (review  maps/graphics  and 
determine  impact  on  routes).  A  substantive  review 
of  the  Weather  Analysis  CCT  is  indicated.  The  test 
does  not  appear  to  represent  an  adequate  sampling 
of  the  relevant  task  and  knowledge  domains  as 
deflned  by  the  available  jobAask  infoimation  for 
purposes  of  content  validation.  The  overall  ratings 
for  the  ‘‘Big  Picture”  weather  analysis  problems,  in 
which  the  students  review  the  continental  U.S. 
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'veather  picture,  sample  approximately  50%  of  the 
relevant  weather  knowledges  and  tasks.  A  substan¬ 
tive  review  of  these  laboratory  problems  is  also 
indicated.  On  the  other  hand,  the  “Route”  weather 
problems,  in  which  students  assess  the  impact  of 
continental  weather  on  specific  routes  of  fli^t,  and 
attempt  to  convey  that  assessment  to  pilots,  sampled 
71%  of  the  defined  knowledges  and  87%  of  the 
delineated  tasks.  The  “Route"  weather  analysis  prob¬ 
lems  appear  to  adequately  sample  the  relevant  do¬ 
mains. 

Summary 

Overall,  8  out  of  10  graded  components  in  the 
FSS  Screen  appear  to  adequately  sample  the  knowl¬ 
edge  and  task  domains  associated  with  particular 
job  functions.  Therefore,  FAA  Academy  course 
50232  may  be  said  to  bear  some  reasonable,  mani¬ 
fest  relationship  to  the  job  of  a  flight  service  special¬ 
ist.  However,  the  review  by  the  field  SME  panel  also 
indicated  specific  areas  of  inadequate  sampling  of 
knowledges  and  tasks,  particularly  in  the  Weather 
Analysis  CST  and  “Big  Picture”  laboratory  prob¬ 
lem. 

RECOMMENDATIONS 

Interim  use  of  the  FSS  Screen 
The  Uniform  Guidelines  (section  5.J)  provides 
that  a  selection  procedure  that  is  not  fully  supported 
by  required  validity  evidence  may  be  used  on  an 
interim  basis  for  making  pass/fail  determinations 
provided 

“(1)  The  user  has  available  substantial  evi¬ 
dence  of  validity,  and  (2)  the  user  has  in  progress, 
when  technic^y  feasible,  a  study  which  is 
designed  to  produce  the  additional  evidence 
required  by  these  guidelines  within  a  reason¬ 
able  time.” 

The  evidence  developed  in  this  report  provides 
substantial  support  for  the  validity  of  the  program  as 
currently  implemented.  However,  that  same  evi¬ 
dence  points  to  specific  weaknesses  that  must  be 
rectified.  Course  50232  might  be  used  as  the  interim 
FSS  screening  tool,  while  revision  of  identified 
weaknesses  are  completed  and  criterion-related  vali¬ 
dation  and  fairness  studies  conducted.  Active  ef¬ 
forts  must  be  made  to  coUea  additional  evidence, 
with  special  attention  to  the  representation  of  mi¬ 
norities  and  women,  about  the  fairness,  validity,  and 
utility  of  the  program  as  well  as  to  clearly  document 


revisions  and  to  gauge  their  probable  effect  during 
any  interim  use.  However,  as  noted  recently  by 
Cronbach  (1988,  p.  5),  validation  is  never  finished; 
revision  and  validation  are  intermeshed  and  proceed 
jointly.  Validation,  as  is  revision  and  development, 
is  an  ongoing  process  of  accumulating  evidence, 
testing explicitand  implicithypo theses  about  people 
and  jobs,  and  making  appropriate  adjustments  to 
both  theory  and  practice  on  the  basis  of  available 
informatioa  This  study  represents  but  one  more 
step  in  that  long  accumulation  of  evidence. 

Notes 

'  This  work  was  performed  under  task  AM-90- 
C-HRR-123.  The  opinions  and  interpretations  ex¬ 
pressed  are  those  of  the  author  alone  and  are  not 
necessarily  those  of  the  FAA. 

^Further  information  with  regard  to  this  study 
can  be  obtained  from: 

Supervisor,  Selection  and  Validation 
Research  Section  (AAM-523) 

Civil  Aeromedical  Institute 
P.  O.  Box  25082 
Oklahoma  City,  OK  73125 
(405)  680-4839;  (FTS)747-4839 

^Rating  materials  are  available  from  the  Super¬ 
visor,  Selection  and  Validation  Research  Section 
(AAM-523).  CAMI. 

*  The  source  data  for  this  study  are  maintained 
in  the  archives  of  the  Selection  and  Validation 
Research  Section  (AAM-523).  An  electronic  data 
base  of  student  scores  in  FAA  Academy  course 
50232  is  underdevelopment.  The  data  base  is  main¬ 
tained  for  research  purposes  only  and  is  confiden¬ 
tial.  No  data  on  an  individual  student  may  be 
released. 

References 

1.  American  Educational  Research  Association. 
Standards  for  educational  and  psychological 
testing.  American  Educational  Research  As¬ 
sociation,  American  Psydbological  Associa¬ 
tion,  &  National  Council  on  Measurement  in 
Education.  Washington  D.C.,  1985. 


15 


2.  Broach,  D.  Briefing  note  to  Minx  Olsea  Vali¬ 
dation  cf  Flight  Service  Initial  Quaiyication 
Program(FAA  Academy  courseit50232).  Qvil 
Aeromedical  Institute,  Human  Resources 
Research  Division,  OklahomaOty,  OK,  1989. 

3.  Chance  v.  Board  of  Examiners,  3  FEP  673, 
1971. 

4.  Cronbach,  L.  J.  Five  perspectives  on  validity 
argument  In  H.  Wainer  &  H.  I.  Braun  (Eds.). 
Test  validity.  Hillsdale,  NJ:  Eribaum,  1988. 

5.  Convey,  J.  J.  The  Flight  Service  Station  Train¬ 
ing  Program:  1981-1985.  Federal  Aviation 
Administration  Office  of  Aviation  Medicine 
Report  No.  DOT/FAA/AM-86/6.  Washing¬ 
ton.  IXT,  1986. 

6.  Convey,  J.  J.  Unpublished  manuscript. 
Attribute  assignments  for  AFSS  tasks.  Civil 
Aeromedical  Institute,  Human  Resources  Re¬ 
search  Division,  1988 

7.  Penn,  E.  D.  Commonwealth  of  Pennsylvania 
V.  O’Neil/,  465  F.Supp.451, 1979. 

8.  Cuesta  v.  State  of  New  York  Office  of  Court 
Administration,  42  EPD  36,949, 1987. 

9.  Equal  Employment  Opportunity  Commission. 
Uniform  guidelines  on  anployee  selection 
procedures.  Federal  Register, 43( 1 66),  38290- 
38315.  August  1978. 

1 0.  Federal  Aviation  Administratioa  FAA  Policy. 
(FAA  Order  KXIO.IA).  Washington,  DC: 
Federal  Aviation  Administration,  1985. 

11.  Federal  Aviation  Administration.  FAA 
Organization:  Headquarters.  (FAA  order 
1 100.2Q.  Washington,  DC:  Federal  Aviation 
Administration,  1989. 

12.  Federal  Aviation  Administration  Academy. 
National  Air  Trcffic  Training  Program  Flight 
Service  Guide,  Instructor  Course  Manage¬ 
ment  Guide:  Indoctrination  (503231 );  Initial 
trainiig  (50232).  CNdahoma  Qty,  OK:  Fed¬ 
eral  Aviation  Administration  Academy,  1989. 


13.  Flight  Service  Instructicxial  Systems  Unit 
AAC-934A.  Course  management  report  on 
FAA  Academy  course  50232,  Flight  Service 
Specialistlni^QuaUficationTraining.  Okla- 
IwmaQty,  OK:  Fli^  Service  Instructional 
Systems  Unit  November,  1990. 

14.  Gee,  W.  F.  Technical  Report:  Automated 
Flight  Service  Station  Specialist  and  Supervi¬ 
sor  JTA.  (Final  Report  submitted  under  FAA 
contract  DTFA01-86-P-0118).  Washington, 
DC:  Federal  Aviation  Administration,  1986. 

15.  Guardians  Association  v.  Civil  Service  Com¬ 
mission,  23  FEP  909, 1980. 

16.  Jones  v.  Human  Resources  Administration, 
12  FEP  265, 1975. 


17.  Manager,  Air  Traffic  Branch,  AAC-930. 
Memorandum  to  Acting  Manager,  Human 
Resources  Research  Division,  AAM-500. 
Information:  Course  50232,  Flight  Service 
Initial  Qualification  Program,  Validation. 
Oklahoma  City,  OK:  FAA  Academy  Air 
Traffic  Branch,  June  1990. 

18.  Manager,  System  Plans  and  Programs  Divi¬ 
sion,  ATR-100.  Memorandum  to  Manager, 
Technical  Training  Division,  APT-300. 
Action:  Job  Task  Analysis  (JTA)  for  Auto¬ 
mated  Flight  Service  Station  (AFSS)  Air 
Traffic  Control  Specialist  (ATCS). 
WashingtonJXT:  Federal  Aviation  Adminis¬ 
tration,  Febniary  1986. 

19.  Manning,  C.  (1988),  Identification  of  AFSS 
knowledges,  skills  and  abilities.  Unpublished 
manuscript 

20.  Myers,  J.,  &  Manning,  C.  (1988),  A  task 
analysis  of  the  Automated  Flight  Service 
StationSpecialistjobanditsapplicationtothe 
development  of  the  Screen  and  training  pro¬ 
gram.  Unpublished  manuscript 


21. Pidael,  E.  W.  Performance  standards  for 
pass-fail  determinations  in  the  national  cur 
trcffic flight  service  station  training  program. 
FAA  Office  of  Aviation  Medicine  Report  No. 
DOT/FAA/AM-79-18.  Washington.  DC., 
1979. 

22.  Society  for  Industrial  and  Organizational 
Psychology,  Inc.  Principles  for  the  validation 
arid  use  cf  personnel  selection  procedures. 
Society  for  Industrial  and  Organizational  Psy¬ 
chology,  Inc.,  Washington,  E)C:  1987. 

23.  Supervisor,  FSS  Revision  and  Development 
Se^on,  AAC-934A.  Memorandum  to  Carol 
Maturing,  Training  and  Skills  Evaluation, 
AAM-127.  Information:  Automated  flight 
service job  taskanatysis.  Oklahoma  City,  OK: 
Federal  Aviation  Administration  Academy. 
December  1987. 

24.  Supervisor,  FSS  Revision  and  Develc^ent 
Severn,  AAC-934A.  Minutes  taken  at  a  con- 
femice  concerning  the  new  JobT ask  Analysis 
for  Automated  Flight  Service  Specialist  atrd 
its  relationship  to  the  new  National  Flight 
Service  Screen  and  Training  Program.  Okla¬ 
homa  Qty,  OK:  FSS  Revision  and  Develop¬ 
ment  Section,  January  1988. 

25.  Supervisor,  FSS  Revision  and  Devdopment 
Se^on,  AAC-934A.  Minutes  taken  at  a  con¬ 
ference  concerning  the  new  JobTask  Analysis 
for  Automated  Flig^  Service  Specialist  and 
its  relationship  to  the  new  National  Flight 
Service  Screen  and  Training  Program.  Okla¬ 
homa  Qty,  OK:  FSS  Revirion  and  Develop¬ 
ment  Section,  Feluuary  1988. 


26.  Thompson,  D.  E.,  &  Theunpson,  T.  A.  Court 
standards  for  job  analysis  in  test  validatioa 
Personnel  Psychology,  35, 865-874, 1982. 


17 


yruA  COVUNMCNT  niNTING  OfTKE!  NM  -  StMlWltMl 


